Skip to content

Avoid unnecessary identifier lookahead - #26525

Merged
charliermarsh merged 1 commit into
mainfrom
charlie/codex-ascii-identifier-lexing
Jul 13, 2026
Merged

Avoid unnecessary identifier lookahead#26525
charliermarsh merged 1 commit into
mainfrom
charlie/codex-ascii-identifier-lexing

Conversation

@charliermarsh

@charliermarsh charliermarsh commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Identifier lexing previously inspected the following one or two characters for a quote before checking whether the identifier could be a string prefix. Gate those lookaheads behind a shared string-prefix check, so ordinary identifiers avoid unnecessary cursor reads while string-prefix flags continue through the existing path.

@astral-sh-bot

astral-sh-bot Bot commented Jul 3, 2026

Copy link
Copy Markdown

Memory usage report

Memory usage unchanged ✅

@astral-sh-bot

astral-sh-bot Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

ecosystem-analyzer results

No diagnostic changes detected ✅

Flaky changes detected. This PR summary excludes flaky changes; see the HTML report for details.

Full report with detailed diff (timing results)

@astral-sh-bot

astral-sh-bot Bot commented Jul 3, 2026

Copy link
Copy Markdown

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

@codspeed-hq

codspeed-hq Bot commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 153 untouched benchmarks
⏩ 4 skipped benchmarks1

Comparing charlie/codex-ascii-identifier-lexing (46205d8) with main (70e2510)

Open in CodSpeed

Footnotes

  1. 4 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@charliermarsh
charliermarsh marked this pull request as ready for review July 12, 2026 20:43
@charliermarsh charliermarsh added the performance Potential performance improvement label Jul 12, 2026
@charliermarsh

charliermarsh commented Jul 12, 2026

Copy link
Copy Markdown
Member Author

Do you still dislike this?

@MichaReiser

MichaReiser commented Jul 13, 2026

Copy link
Copy Markdown
Member

I still think the next_token change increases complexity. Would you mind splitting the PR into two, so that we can attribute the performance win to each improvement separately.

@charliermarsh
charliermarsh force-pushed the charlie/codex-ascii-identifier-lexing branch from d3f97b7 to 46205d8 Compare July 13, 2026 11:16
@charliermarsh charliermarsh changed the title Avoid redundant lexer work Avoid unnecessary identifier lookahead Jul 13, 2026
@charliermarsh charliermarsh mentioned this pull request Jul 13, 2026
Merged
@charliermarsh
charliermarsh merged commit 6fd508a into main Jul 13, 2026
62 checks passed
@charliermarsh
charliermarsh deleted the charlie/codex-ascii-identifier-lexing branch July 13, 2026 13:52
@BrewTestBot BrewTestBot mentioned this pull request Jul 16, 2026
Merged
@scottames-github-bot scottames-github-bot Bot mentioned this pull request Jul 21, 2026
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MichaReiser MichaReiser MichaReiser approved these changes
@dhruvmanila dhruvmanila Awaiting requested review from dhruvmanila dhruvmanila is a code owner

Assignees

No one assigned

Labels

performance Potential performance improvement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@charliermarsh @MichaReiser